Migrated MariaDB, MySQL, PostgreSQL tests to use testcontainers to simplify local setup - #2011
Conversation
| st.setTime(16, java.sql.Time(System.currentTimeMillis())) | ||
| st.setTimestamp(14, SqlTimestamp(System.currentTimeMillis())) | ||
| st.setTimestamp(15, SqlTimestamp(System.currentTimeMillis())) | ||
| st.setTime(16, SqlTime(System.currentTimeMillis())) |
There was a problem hiding this comment.
Automatic refactorings break such fully qualified links :( I recovered it as import alias, should survive file moves etc
There was a problem hiding this comment.
🟡 Not ready to approve
It currently introduces a dependency-resolution issue in the version catalog and a Kotlin compilation error from an unused import.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR migrates the JDBC integration tests for MariaDB/MySQL/PostgreSQL to Testcontainers so contributors can run them locally with Docker, while keeping CI’s default test task free of Docker-dependent tests.
Changes:
- Added Testcontainers dependencies to the version catalog and
dataframe-jdbctest configuration. - Updated MariaDB/MySQL/PostgreSQL test suites to start/stop containers in
@BeforeClass/@AfterClassrather than relying on localhost DBs. - Added a dedicated
testcontainersTestGradle task and excluded the Testcontainers package from the defaulttesttask.
File summaries
| File | Description |
|---|---|
| gradle/libs.versions.toml | Adds Testcontainers version + catalog entries for DB modules. |
| dataframe-jdbc/build.gradle.kts | Adds Testcontainers test deps; excludes Docker tests from test and introduces testcontainersTest. |
| dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/postgresTest.kt | Switches Postgres tests to a managed PostgreSQLContainer. |
| dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/postgresConnectionUrlTest.kt | Reworks Postgres URL parsing tests to run against a container. |
| dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/mysqlTest.kt | Switches MySQL tests to a managed MySQLContainer. |
| dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/mariadbTest.kt | Switches MariaDB tests to a managed MariaDBContainer. |
| dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/images.kt | Centralizes Docker image tags used by the Testcontainers tests. |
Review details
Comments suppressed due to low confidence (2)
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/mysqlTest.kt:20
- Unused import
org.junit.Ignorewill fail Kotlin compilation (unused imports are errors). The@Ignoreannotation was removed, so this import should be removed too.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/postgresTest.kt:30 - Unused import
org.junit.Ignorewill fail Kotlin compilation (unused imports are errors). The@Ignoreannotation was removed, so this import should be removed too.
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| testcontainers-postgresql = { group = "org.testcontainers", name = "testcontainers-postgresql", version.ref = "testcontainers" } | ||
| testcontainers-mysql = { group = "org.testcontainers", name = "testcontainers-mysql", version.ref = "testcontainers" } | ||
| testcontainers-mariadb = { group = "org.testcontainers", name = "testcontainers-mariadb", version.ref = "testcontainers" } |
|
https://gh.io/copilot-coding-agent-docs will this work on Windows or on Team City? |
|
Did you test it with it already on TC? See https://www.jetbrains.com/help/teamcity/2026.1/gradle.html?TeamCity%20Documentation#Container+Settings and https://www.jetbrains.com/help/teamcity/2026.1/container-wrapper.html?TeamCity%20Documentation (locally, they seem to run well, btw!) |
bad copilot : ( why did it have to commit things to my branch and break CI. Besised, i'd do abstract test class and have LocalMariaDb tests + DockerMariaDb tests instead |
On TeamCity eventually yes, need to configure teamcity to run on agents that have docker daemon |
89fcba0 to
7af1db1
Compare
|
Great job! Will test it next week on both MacOS and WIndows, locally and not, and it will be shipped! |
|
Is it possible to migrate MSSQL ? Also, it's very painful to test |
i believe so, but i want to make sure it's ok for us because we need to accept some license agreement to use mssql if overall we're ok with migrating to testcontainers i'll proceed with mssql Due to licencing restrictions you are required to accept an EULA for this container image. To indicate that you accept the MS SQL Server image EULA, call the acceptLicense() method, or place a file at the root of the classpath named container-license-acceptance.txt, e.g. at src/test/resources/container-license-acceptance.txt. This file should contain the line: mcr.microsoft.com/mssql/server:2017-CU12 (or, if you are overriding the docker image name/tag, update accordingly). Please see the microsoft-mssql-server image documentation for a link to the EULA document. |
zaleslaw
left a comment
There was a problem hiding this comment.
Include in the Readme.md information and links about requirement to set up the Docker env for test conteainers with minimum technical requirements for 3 OS or link to that
- describe the idea of test separation (it's complicated)
`## :dataframe-jdbc
This module, published as dataframe-jdbc, contains all logic and tests for DataFrame to be able to work with
JDBC data sources.
See Read from SQL databases for more information
about how to use it.
Testing
The JDBC tests are split into three tiers by how they obtain a database. This keeps the default build
fast and dependency-free, while still allowing full coverage against real database engines.
1. H2 tests (default suite)
These run against an in-memory H2 database using its compatibility modes
(MODE=PostgreSQL, MODE=MySQL, MODE=MariaDB, MODE=MSSQLServer, etc.). They need no Docker and no external
server, so they are part of the regular test task and run in CI on every build.
Location: src/test/kotlin/.../io/h2/ (e.g. PostgresH2Test, MysqlH2Test, MariadbH2Test, MssqlH2Test).
Together with the SQLite/DuckDB tests they form the baseline coverage that always runs.
./gradlew :dataframe-jdbc:test2. Testcontainers tests (real DB engines via Docker)
These spin up real PostgreSQL, MySQL, and MariaDB engines in Docker using
Testcontainers. They give the highest-fidelity coverage but require a running
Docker daemon, so they are excluded from the default test task and run under a dedicated Gradle task.
Location: src/test/kotlin/.../io/testcontainers/ (package org.jetbrains.kotlinx.dataframe.io.testcontainers).
The container image tags are configured in build.gradle.kts via BuildConfig
(POSTGRES_IMAGE, MYSQL_IMAGE, MARIADB_IMAGE).
Requirements: a running Docker daemon.
./gradlew :dataframe-jdbc:testcontainersTest3. Local DB tests (servers on localhost)
These connect to database servers you run yourself on localhost (e.g. Postgres on 5432, MySQL, MariaDB, MSSQL).
They are meant for local, manual verification against a specific server and are excluded from the default
test task.
Location: src/test/kotlin/.../io/local/. Only classes whose names end with LocalTest
(PostgresLocalTest, MySqlLocalTest, MariadbLocalTest, PostgresConnectionUrlLocalTest) are picked up by the
dedicated task. The connection URLs/credentials are hardcoded in each test class, so start a matching server
before running them.
./gradlew :dataframe-jdbc:localDbTestHow the exclusions are wired
The default test task explicitly excludes both the Testcontainers and local-DB tests, and each is exposed as its
own task (see build.gradle.kts):
private val testcontainersTests = "org.jetbrains.kotlinx.dataframe.io.testcontainers.*"
private val localDbTests = "org.jetbrains.kotlinx.dataframe.io.local.*LocalTest"
tasks.test {
filter {
excludeTestsMatching(testcontainersTests)
excludeTestsMatching(localDbTests)
}
}
tasks.register<Test>("testcontainersTest") { /* includeTestsMatching(testcontainersTests) */ }
tasks.register<Test>("localDbTest") { /* includeTestsMatching(localDbTests) */ }So: H2 (and SQLite/DuckDB) run by default; Testcontainers and local-DB tests are opt-in via their own tasks.`
|
Yes, we can proceed with MSSQL Testcontainers.
Our purpose is clear: use MSSQL only for automated integration testing. |
|
Could help with, but need to verify #563 |
ffe95cc to
fb4e902
Compare
Tests still will not be run on CI for now. But it'll be now possible to run MariaDB, MySQL, PostgreSQL in one click, providing you have docker installed on PC. It's the only prerequisite.
Let's start with those and migrate imbd, mssql example after
For reference. Looks like their JUnit4 examples are a bit outdated and nowadays modern "container" classes do not implement "testrule" class, so we use manual lifecycle management instead
https://java.testcontainers.org/modules/databases/mysql/
https://java.testcontainers.org/modules/databases/mariadb/
https://java.testcontainers.org/modules/databases/postgres/